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(57) A selected image view generator is disclosed 
for generating a selected portion of a larger compressed 
image. A selected image view from an overall image is 
generated with significantly reduced computation and 
reduced latency (for video processing). Frame insensi- 
tivity permits selected image views to be formed from 
static images or from a video stream. Multiple users can 
simultaneously control a selected view received from an 
image source. Remote cameras can be panned elec- 
tronically, rather than physically. The overall com- 
pressed image may be encoded using a suitable intra- 
frame macrob lock-based image encoder, provided that 
each macroblock is encoded independently. Each mac- 
roblock optionally contains a macroblock identifier indi- 
cating the position of the macroblock in a given image. 
Each transmitted overall image and selected image 
view optionally includes a frame header indicating the 
number of macroblocks in the transmitted image. 
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Description 

Field of the Invention 

[0001] The present invention relates to image encod- 
ing and decoding techniques, and more particularly, to 
a method and apparatus for generating a partial image 
from a larger compressed image (or a plurality of indi- 
vidual images). 

Background of the Invention 

[0002] Due to the limited bandwidth of transmission 
channels, there are a limited number of bits available for 
encoding image information, such as image information 
generated by a camera for transmission to one or more 
remote users. Thus, there are many image encoding 
techniques available which encode the image informa- 
tion with as few bits as possible using compression tech- 
niques, while still maintaining the quality and intelligibil- 
ity that are required for a given application. 
[0003] Remote cameras, such as those used for se- 
curity applications, traffic monitoring or daycare moni- 
toring, are typically panned by physically moving the 
camera. I n addition to the possibility of a mechanical fail- 
ure, the utility of such remote cameras is limited in that 
only one user can control the camera at a time. For multi- 
user applications, however, such limited user control of 
the camera view is not practical. A number of software 
techniques have been developed for permitting a 
number of users to view selected portions of a larger 
image (or a composite image generated from a plurality 
of individual images). 

[0004] Permitting multiple selected views of a larger 
image, however, becomes more difficult if the larger im- 
age is compressed. Specifically, since image data fol- 
lowing image compression is of variable length, pixel 
boundaries are not readily detectable in a compressed 
image. In addition, since many encoding techniques ex- 
hibit intra-frame pixel dependencies, such as encoding 
the difference values for adjacent DC coefficients under 
the JPEG standard, the pixel values must be modified 
when generating a selected portion of a larger image, 
to reflect the reordering of the subset of pixels in the 
selected image view. 

[0005] Typically, when, generating a selected portion 
of a larger compressed image, the larger image must 
be decompressed into the pixel domain, before the pixel 
values are reordered and assembled to create each of 
the selected image views. Thereafter, each of the se- 
lected image views are compressed to form the final im- 
ages transmitted to each user. The more popular image 
compression techniques, such as JPEG and MPEG, 
typically perform three steps to generate a compressed 
image, namely, (i) transformation, such as a discrete co- 
sine transform (DCT), (ii) quantization; and (iii) run- 
length encoding (RLE). Likewise, to decompress imag- 
es using these same image compression techniques, 



the inverse of the compression steps are performed by 
the receiver on the compressed image, namely, (i) run- 
length decode; (ii) dequantization; and (iii) inverse dis- 
creet cosine transform (I DCT). 
s [0006] Thus, to create N selected image views from 
a larger compressed image, conventional techniques 
require one image decompression, N pixel reorde rings, 
and N compressions. 

10 Summary of the Invention 

[0007] Generally, a selected image view generator for 
generating a selected portion of a larger compressed 
image is disclosed. The larger compressed image in- 

is eludes a plurality of macroblocks of image data, encod- 
ed using an intraf rame encoding technique that encodes 
the macroblocks independently, thereby allowing mac- 
roblocks within the selected image view to be selectively 
assembled to create the selected image view. The se- 

20 lected image view generator stores the larger com- 
pressed image, receives an indication of the selected 
image view from a user, identifies the macroblocks from 
the larger image included in the selected image view 
and assembles the identified macroblocks to form the 

25 selected image view. 

[0008] According to another aspect of the invention, 
multiple users can simultaneously control a selected 
view received from an image source. The overall image 
may include one or more static or real-time images. The 

30 selected image from a larger overall image may be 
used, for example, with a 360° panning camera, to per- 
mit each user to select a desired real-time or static view. 
For example, for a tennis match, different users can 
watch different players from the same video feed. 

35 [0009] The larger compressed image may be encod- 
ed using a suitable intra-frame macroblock-based im- 
age encoder, provided that each macroblock is encoded 
independently to ensure that the correlation between 
DC coefficients is restricted to within a given macrob- 

40 lock. Independent macroblocks may be achieved within 
the JPEG standard, for example, by initiating the Restart 
interval for each macroblock. Each macroblock option- 
ally contains a macroblock identifier. Each macroblock 
identifier initially indicates the position of the macroblock 

45 in a given image. The macroblock identifiers are option- 
ally renumbered for each selected image view to indi- 
cate the position of the macroblock in the selected im- 
age view. In addition, each transmitted overall image 
and selected image view optionally includes a frame 

so header indicating the number of macroblocks in the 
transmitted image. 

Brief Description of the Drawings 

55 [0010] 

FIG. 1 A illustrates a block diagram of a network suit- 
able for transferring information from an image 
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source device to one or more destination devices; 
FIG. 1 B illustrates a selected image view from an 
overall larger image; 

FIG. 2 illustrates one embodiment of the selected 
image view generator of FIG. 1 ; and s 
FIG. 3 is a flow chart describing the selected image 
view generation process of FIG. 2. 

Detailed Description 

[0011] FIG. 1 A shows an illustrative network environ- 
ment 100 for transferring multimedia information, such 
as image, video, voice or data information, or a combi- 
nation of the foregoing, between one or more sources 
and destinations, such as end users 110-113 and an im- 
age source 130. As shown in FIG. 1B, each end user 
110-113, such as end user 110, desires to view a select- 
ed portion 150 of a compressed image 160. The com- 
pressed image 160 is generated by one or more image 
sources, such as image source 130. The image source 
130 may be embodied, for example, as a 360° camera 
that produces panoramic images, or one or more cam- 
eras generating images that may be combined into a 
composite image. For a detailed discussion of a suitable 
360° camera, see United States Patent Application Se- 
rial Number 08/433,356, entitled "Panoramic Viewing 
Apparatus," assigned to the assignee of the present in- 
vention and incorporated by reference herein. 
[0012] A selected image view generator 200, dis- 
cussed further below in conjunction with FIG. 2, gener- 
ates the selected portion (s) 150 of the image 160, 
shown in FIG. 1B. As used herein, the image 160 in- 
cludes a composite image generated from one or more 
individual images. In one embodiment, each user re- 
ceives the overall image 160, as well as the currently 
selected image view 1 50 The transmitted overall image 
160 provides the user 110 with an orientation and per- 
mits the user 1 1 0 to select the desired view 1 50. For one 
set of video applications, the overall image 160 can be 
transmitted at a rate below the standard frame rate, and 
the smaller selected image view 150 can be transmitted 
at the highest rate permitted by the available bandwidth. 
In one video implementation, the user selects a view 
point, and whatever passes in the selected viewpoint 
over time is presented to the user In a still image imple- 
mentation, the smaller selected image view 150 can be 
presented with a higher resolution than the overall im- 
age 1 60 and/or mapped to a larger portion of the display 
screen. 

[0013] The selected image view generator 200 may 
utilize a variety of image inputs encoded with independ- 
ent macroblocks, for example, received from a file serv- 
er, a video-on-demand system or a real-time camera. 
The network environment 1 00 may be embodied, for ex- 
ample, as the Public Switched Telephone Network 
("PSTN"), the internet, or other broadband networks, as 
would be apparent to a person of ordinary skill. In the 
illustrative embodiment, the end users 110-113 can uti- 



lize work stations (not shown), or other general-purpose 
computing devices. 

[001 4] According to a feature of the present invention, 
a selected image view 1 50 from an overall image 160 is 
generated with significantly reduced computation and 
reduced latency (for video processing), permitting view- 
ing of live events without noticeable delay. In addition, 
the present invention permits selected image views 150 
to be formed from static images or from a video stream 
or both. According to a further feature of the present in- 
vention, multiple users can simultaneously control a se- 
lected view received from the image source 1 30. in this 
manner, a user 110 obtains personalized access to the 
image source 130. 

[0015] The image 160 may be encoded using a suit- 
able intra-frame mac rob lock-based image encoder, 
such as a JPEG image encoder or an MPEG intra-frame 
image encoder, as modified herein to incorporate the 
features and functions of the present invention. In the 
illustrative JPEG embodiment, each image 160 consists 
of macroblocks of pixels. Macroblocks are typically a 1 6 
xSora 16x 16 pixel array. Typically, macroblocks are 
transmitted sequentially from left to right, top to bottom. 
Thus, for example, 2400 16x8 macroblocks would con- 
stitute an entire 640 x 480 pixel frame. The macroblocks 
constitute one or more independent color components, 
which may be at different resolution levels. In the illus- 
trative JPEG embodiment, each illustrative 16x8 mac- 
roblock is represented by 2 luminance (Y) blocks and 2 
chrominance (C b ,C r ) blocks, with each block containing 
I DC coefficient value, and 63 AC coefficient values. 
[0016] Under the JPEG standard, the AC and DC co- 
efficients are coded in a different manner. For a detailed 
discussion of JPEG encoding, see William B Pennebak- 
er and Joan L. Mitchell, JPEG Still Image Data Com- 
pression Standard (Ban Nostrand Reinhold, 1 993), in- 
corporated by reference herein. Generally, the DC co- 
efficient is a multiple of the average value in the 8x8 
block. Since the average pixel value in any 8x8 block 
will not differ substantially from the average value in a 
neighboring block, the DC coefficient values will typical- 
ly not vary significantly. Thus, in order to achieve further 
bit rate reductions in the JPEG standard, the differences 
between adjoining coefficients are typically encoded 
rather than the coefficients themselves. 
[001 7] In accordance with the present invention, how- 
ever, each macroblock is encoded independently. Thus, 
the correlation between DC coefficients is restricted to 
within a given macroblock. Independent macroblocks 
may be achieved within the JPEG standard, for exam- 
ple, by initiating the Restart interval for each macrob- 
lock. One suitable system for encoding the macroblocks 
independently is the Montage™ video coding system, 
being developed by Lucent Technologies Inc. Holmdel, 
NJ. For a detailed discussion of the Montage™ video 
coding system, see R. GaglianelloandG.L Cash, "Mon- 
tage: Continuous Presence Teleconference Utilizing 
Compressed Domain Video Bridgin," I.E.E.E. I.C.C. 
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(June, 1995). 

[0018] Each macroblock optionally contains a mac- 
roblock identifier. Each macroblock identifier includes 
data representative of the position of the macroblock in 
the image 160. Macroblock identifiers are typically as- 
signed sequentially left to right, top to bottom in each 
image. The macroblock identifiers are generally specific 
to each image 160. In some compression standards, 
such as JPEG, macroblock identifiers are generally not 
utilized. In accordance with the present invention, the 
macroblock identifiers associated with macroblocks in 
a selected image view 150, if present, are optionally re- 
numbered to indicate the position of the macroblock in 
the selected image view 150. In addition, each image 
frame optionally includes a frame header indicating the 
number of macroblocks in the transmitted image. 
[0019] FIG. 2 is a block diagram showing the archi- 
tecture of an illustrative selected image view generator 
200. The selected image view generator 200 preferably 
includes a processor 210 and related memory, such as 
a data storage device 220. The processor 210 may be 
embodied as a single processor, or a number of proc- 
essors operating in parallel. The data storage device 
220 or other memory are operable to store one or more 
instructions, which the processor 210 is operable to re- 
trieve, interpret and execute. 

[0020] The data storage device 220 preferably in- 
cludes an area of memory 230 for storing the com- 
pressed image(s) 160. The selected image view gener- 
ator 200 may receive the compressed image(s) 160 
from the image source 130 for processing in real-time 
or static compressed images may be retrieved from a 
database of images, such as a video server, as would 
be apparent to a person of ordinary skill. In addition, as 
discussed in conjunction with FIG. 3, the data storage 
device 220 preferably includes a selected image view 
generation process 300. Generally, the selected image 
view generation process 300 (i) retrieves the encoded 
input image data; (ii) retrieves an indication of the image 
view selected by the user; and (iii) generates and trans- 
mits the selected image view 150 to the user. 
[0021] As shown in FIG. 3, the selected image view 
generation process 300 initially retrieves the image 160 
that has been compressed using a mac rob lock-in de- 
pendent image encoder (not shown) during step 310. 
The selected image view generation process 300 then 
receives an indication of the selected image view from 
the user, such as user 110, during step 320. The user 
may indicate the selected image view, for example, us- 
ing a graphical user interface (GUI ) to outline the desired 
region of a larger image displayed on a computer 
screen. The outline box created by the user using the 
GUI can optionally snap to the dimensions of the mac- 
roblocks, in a known manner. 

[0022] Thereafter, the selected image view genera- 
tion process 300 identifies the macroblocks from the 
overall image 160 during step 330 that are included in 
the selected image view 150. In an embodiment where 



each macroblock contains a macroblock identifier (for 
example, when the images are encoded by an MPEG 
encoder), the macroblock identifiers associated with the 
macroblocks identified during step 330 are adjusted dur- 
5 ing step 340 to indicate the position of each macroblock 
in the selected image view 150. Finally, the selected im- 
age view 1 50 is transmitted to the user, such as user 
110, during step 350, before program control terminates 
during step 360. The transmitted image preferably in- 
fo eludes a header indicating the number of macroblocks 
in the selected image view 1 50. 
[0023] It is to be understood that the embodiments 
and variations shown and described herein are merely 
illustrative of the principles of this invention and that var- 
15 ious modifications may be implemented by those skilled 
in the art without departing from the scope of the inven- 
tion. 



20 Claims 

1 . A method of generating a selected image view from 
a larger compressed image, said larger com- 
pressed image being made up of at least a pluality 

25 of macroblocks of image data, said method com- 
prising the steps of: 

identifying the macroblocks from the larger 
compressed image included in the selected im- 
30 age view, said larger compressed image en- 

coded using an intraf rame encoding technique 
that encodes said macroblocks independently; 
and 

assembling said identified macroblocks to form 
35 said selected image view. 

2. The method according to claim 1 , further compris- 
ing the step of including a header with said selected 
image view indicating the number of said macrob- 

40 locks in said selected image view. 

3. The method according to claim 1 or claim 2, wherein 
each of said macroblocks has an associated mac- 
roblock identifier indicating the position of said mac- 

45 roblock in an image. 

4. The method according to claim 3, further compris- 
ing the step of adjusting said macroblock identifier 
to indicate the position of each macroblock in the 

so selected image view. 

5. The method according to any of the preceding 
claims wherein said intraf rame encoding technique 
is JPEG. 

55 

6. The method according to any of claims 1 to 4, 
wherein said intraframe encoding technique is 
MPEG. 
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7. The method according to any of the preceding 
claims, wherein said larger compressed image is a 
panoramic view. 

8. The method according to any of the preceding 
claims, wherein said larger compressed image is 
one image frame in a video stream. 

9. The method according to claim 8, wherein the con- 
tent in said selected image view can change with 
each frame of said video stream. 

10. The method according to any of the preceding 
claims, wherein said identifying step comprises the 
step of identifying the macroblocks from the larger 
compressed image included in the image view se- 
lected by a user using a graphical user interface to 
outline the desired region of the larger compressed 
image. 

11. The method according to any of the preceding 
claims, further comprising the step of decoding said 
larger compressed image only enough to detect 
said macroblocks. 

12. The method according to any of the preceding 
claims, wherein said independent macroblocks are 
obtained by restarting the encoding of said larger 
compressed image between each of said macrob- 
locks. 

1 3. A method as claimed in any of the preceding claims 
including an initial step of encoding said macrob- 
locks independently using said intraframe encoding 
technique. 

14. The method according to any of the preceding 
claims, further comprising the step of transmitting 
said selected image view to a user. 

15. A selected image view generator for generating a 
selected image view from a larger compressed im- 
age, said larger compressed image being made up 
of at least a plurality of macroblocks of image data, 
comprising: 

a data storage device for storing said larger 
compressed image, said larger compressed 
image being encoded using an intraframe en- 
coding technique that encodes said macrob- 
locks independently: 

an input for receiving an indication of said se- 
lected image view from a user; and 
means arranged to carry out a method as 
claimed in any of claims 1 to 12. 

16. The selected image view generator according to 
claim 16, further comprising a communications port 
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for transmitting said selected image view to a user. 
17. An image receiver, comprising: 



s a transmitter for transmitting a request for a se- 

lected image view of a larger compressed im- 
age; 

a receiver for receiving said selected image 
view, said selected image view comprised of a 
10 plurality of independent macroblocks encoded 

using an intraframe encoding technique; and 
decompression means for decompressing said 
macroblocks and presenting said selected im- 
age view to a user. 

15 

18. A method ot transmitting a selected image view 
from a larger compressed image to a user, said larg- 
er compressed image being made up of at least a 
plurality of macroblocks of image data, said method 
20 comprising the steps of: 

transmitting said larger compressed image to a 
user at a first rate; 

receiving a selection of a selected image view 
25 within said larger compressed image from a us- 

er; 

identifying the macroblocks from the larger 
compressed image included in the selected im- 
age view, said macroblocks being independ- 
30 ently encoded using an intraframe encoding 

technique; and 

transmitting said identified macroblocks to said 
user at a second rate to form said selected im- 
age view, said second rate being faster than 
35 said first rate. 
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